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PROFILING COMPLETED RESEARCH 



The evaluation of the quality of completed research in 
education has two distinct components. The first of these 
components is the problem attacked in the study. The second is 
the methodological adequacy of the study conducted. Profiling, 
the procedure described in this paper, deals with the latter, the 
evaluation of the methodological adequacy. It avoids evaluation 
of the problem itself on the belief that the importance of a given 
problem can only be established through ah historical perspective. 
To assert otherwise would imply the existence of a preferred 
value system. 

Evaluation of methodological adequacy of a given piece of 
research is a prerequisite for the acceptance or rejection of the 
conclusions of that investigation. Such conclusions can be no 
stronger than the methods utilized in generating and analyzing 
the data on which the conclusions are reached. In the past we 
have operated on the assumption: if the methodology is sound, 
the conclusion can be accepted and vice versa. The faultiness of 
this assumption is one of the problems that have long plagued 
both the improvement of and use of educational research. 

Research methodology is multifaceted. It involves an inherent 
logical argument, the selection of subjects to be studied, 
structuring of experiences for those subjects, measurement, and 
the analysis of the generated data. It is possible to have sound 
procedures in some of these facets and weak procedures in others; 
a possibility that precludes a statement that a conclusion is based 
either on sound or unsound methods. 

The problem is further complicated. Needs for surety in 
varying times and professional circumstances set the quality 
standard for research methods. If the need for knowledge in an 
area is great: the methodological development crude, and the 
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amount of risk to personal safety low. conclusions can be 
accepted and operated on despite weaknesses in their methodo- 
logical base. In another set of circumstances this would be wholly 
unacceptable. Since the use to which a conclusion might be put 
cannot be controlled, an absolute level of quality cannot be 
established for each research effort. 

Regardless of the knowledge needs or professional circum- 
stances, a given conclusion ought not to be accepted, held 
tentatively or rejected without evaluation of the research 
methods underlying it. It is asserted that the profiling procedure 
described in this paper will facilitate the labeling of the 
methodology of completed research reports. When this labeling 
has been completed, the user of that study can make sounder 
decisions regarding the acceptance or rejection of its conclusions. 

ELEMENTS IN PROFILING 

In conducting an empirical study an investigator does numer- 
ous things. Those things are the elements on which the profiling 
activity focuses. They include: (I) the structuring of a logical 
argument; (2) the generation of data; and (3) the analysis of that 
data. All three items are involved in investigations which test 
hypotheses while only items two and three are used in studies 
which attempt to answer empirical questions. 

THE INHERENT LOGICAL ARGUMENT is of crucial im 
portance when a study attempts a test of a hypothesis. In effect, 
the investigator is trying to determine the truth or falsity of his 
hypothesis. He does this through a logical argument described by 
Poly a. f It consists of a major premise, one or more minor 
premises, and a conclusion. 

The major premise is typically a statement which asserts, “If 

the - _ _ hypothesis is a true statement; then 

.events will be observed as indicators of that truth.” 

An example of a major premise can be seen in a study reported a 
few years ago by McNeil.2 He proposes a hypothesis which asserts 



that teachers present different instructional treatments for the 
two sexes of their students. As indicators of the truth of that 
statement he reasoned that boys would be nominated more often 
than girls as recipients of certain kinds of teacher action. His 
major premise could be stated as, 

If the hypothesis (teachers provide different instructional 
treatment for boys than they do for girls) is a true 
statement, then systematic differences by sex will be seen 
when children are asked to name the students who receive 
specified teacher treatments. 

Two kinds of minor premises have been evolved from Polya’s 
work by Raths.3 The first of these deals with the predicted 
observation. Was it or was it not seen? The premise’s exact nature 
in a given study is determined after the data are analyzed. In the 
McNeil example used above, significant differences by sex were 
observed. The minor premise in that case would be, “There is a 
systematic sex differentiation in the nominations.” 

The second category of minor premises deals with rival 
hypotheses, rival or alternative explanations for the observation 
reported in the first minor premise. The premise is based upon 
the recognition that an effect in the social sciences often has 
multiple causes. Once an observation has been made, all its 
possible causes must be examined before it can be concluded that 
the observation supports the truth of a specific hypothesis. One 
of three general conditions might exist ranging from no rival 
hypotheses are apparent to rival hypotheses may exist to rival 
hypotheses are definitely involved. 

The final element of the logical argument is theiconclusion. Its 
form in a given study is dependent \upon the nature of the two 
minor premises. From the first minor premise comes information 
as to whether or not the truth of the hypothesis being tested is 
supported. If the consequents predicted are observed, support for 
the truth of the hypothesis is presented. If the observation is not 
made, support cannot be claimed. (Note: Failure to make the 
predicted observation does not automatically mean rejection of 
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the hypothesis.) The second minor premise determines the 
strength of the conclusion. If rival hypotheses arc known to be 
present very weak support for the truth of the hypothesis has 
been developed. If there is the possibility but not the probability 
of rival hypotheses, tentative support is generated. And finally, if 
no rival hypotheses are conceivable, it is credible that the 
hypothesis is a true statement. 

THE GENERATION OF DATA, the second major facet in 
profiling, involves evaluation of three aspects of data generation: 
units studied; treatments experienced by those units; and 
measurement. If variation in any of these three occurs a different 
set of data is generated. For example, consider an investigation of 
the effects of test anxiety on achievement. If the study 
concentrates on a randomly selected group of high school seniors 
as subjects, one set of data will be generated. If a group of 
students who are divergent on a measure of test anxiety is 
selected as subjects, a different set of data will be generated. 
Given a specified group as subjects, variation in the treatment or 
of their experiences will cause different sets of data to be 
generated. Again the test anxiety problem provides an example. 
One set of data could be generated by a treatment in which the 
subjects are given information about the importance of a test and 
administered a test that is constructed for students at a much 
higher level of education than are the subjects. Still a different set 
of data will be generated if the students are repeatedly given a 
test that is very difficult. If the effects of a specific treatment on 
a specific group are measured by a paper and pencil test such as 
Sarazon’s Test of Test Anxiety, one set of data would be 
generated. On the other hand if the seats in the classroom were 
wired and a galvanic skin response measure were taken, quite a 
different set of data would be generated. 

These three aspects of data generation are displayed graph- 
ically in Figure 1.4 The scale of unit quality or representativeness 
runs along the dimension OA, treatment quality OC, and 
measurement quality OG. A project which selected a sample 
perfectly representative of a population of interest would be 
located at Point A. on the cube. If, in that same study, a 
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thorough programing of the content and sequence of the 
treatments was employed in generating data, the project would be 
conceptualized as being at Point B. on the quality cube. Finally, 
if our study employed perfectly objective, valid, and reliable 
measuring techniques, it would be" located at Point E. on the 
cube. 

A given study seldom reaches this level of data generation 
quality. Rather it falls somewhere between the extremes. To 
facilitate profiling ordinal scales have been developed for these 
three dimensions as shown below. 



Dimensions for the Research Quality Cube 

Representativeness 

R5= The entire population was studied 

R4= Random selection from a specified population was 
employed to determine which units were studied. 

R3= Purposive sampling from a specified population estab- 
lished the group studied. 

R2= Volunteers were studied. 

Rj = An unidentified group of subjects was studied. 

Treatment 

T$ = A theoretically based treatment, was administered and 
described and controls were employed for mediating 
variables identified in the theory AND for variables 
extraneous to the theory that might have an effect. 

T5 = Same as Ttf with the exception of the lack of controls 
for extraneous variables. 

T4 = Same as T& with the exception of the lack of controls 
for theory encompassed mediating variables and ex- 
traneous variables. 

T3 = No theory stated but the employed treatment described 
in detail sufficient for replication. 

T2 = Commonly known treatment administered but not 
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THE DATA QUALITY CUBE 






Entire population 



reliable, and 

objective 

instrument 
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described in detail. i 

Ti * Something of an undescribed nature was experienced ; 

by the units studied. i 

Measurement S 

Ms= Data were generated through the use of either a j 

commercially standardized or ad hoc instrument AND { 

data are presented which establish high validity and j 

reliability for its use in this measurement task. \ 

M 4 = Data generated through the use of a commercially * 

standardized instrument and evidence presented indicat- j 

ing moderate validity and reliability for this application. • 

M3= Data generated through a commercially standardized 

test but no evidence presented as to its validity and j; 

reliability for this application. i 



M2= Data generated through an ad hoc instrument and \ 

evidence of moderate validity and reliability presented. ) 

M]= Data generated through an ad hoc instrument with j 

either no supporting evidence as to validity and i 

reliability or evidence indicating poor validity and { 

reliability on' either a commercially standardized or ad j 

hoc instrument. { 



DATA ANALYSIS PROCEDURES are the final element in j 

profiling. When data, typically in the form of. numbers, are j 

generated as the ; supporting evidence for a conclusion, under- « 

standing of the meaning of those numbers is incumbent upon the f 

researcher' and the research utilizer. That meaning is not readily J 

apparent if there is a large quantity of numbers. Simplifying 1 

procedures have been developed; procedures wliich are not | 

appropriate for al! kinds of data. j 

The determination of the correct procedure in a given study is | 

not an exact science. In developing a procedural flow chart for | 

the profiling of educational research, sixteen schemes were 
identified which were supposed to assist in the selection of the 
correct statistic for given sets of data. Some of these were 
incomplete schemes in that they purported to deal only with 
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limited kinds of statistical analysis.^ Some imply a comprehens^ 
iveness but fail to be definitive as they list a number of statistics 
appropriate for a given set of conditions.^ 

Since a single comprehensive grid or table for selecting the 
correct analytic procedure could not be found a second task was 
undertaken. Existing statistical procedures were catalogued and 
the assumptions underlying them were listed. An effort to build a 
comprehensive selection procedure by analyzing these items has 
to this point been unsuccessful. (A colleague at Indiana Univer- 
sity? has just recently attacked this problem using Guttman’s 
Facet design and Analysis Technique# with initially promising 
results.) 

Because of these problems three grids have been generated for 
profiling the data analysis procedures. The first of these deals 
with analytic procedures for sample description. It includes 
measures of central tendency and dispersion and classifies the 
procedures by levels of measurement, i.e., nominal, ordinal, and 
interval-ratio. The second grid is used when an associational 
analysis is desired. li has identical labels for its rows and columns 
which refer to the nature of the measurement on the two 
variables to be correlated. The categories in this case are: 

1. Continuous variables (age, height, I.Q., achievement, etc.) 

2. Forced dichotomy (number of persons over and under 100 
I.Q., number of persons weighing over and under 150 
pounds, etc.) 

3. True dichotomy (student-nonstudent, male-female, etc.) 

Given the nature of the two variables on which an associational 
analysis is desired the grid can be used to select the appropriate 
statistic. Four special cases exist and are shown with the grid. 
Three of these are instances in which more than two variables are 
involved. The final case covers correlation among ordinal vari- 
ables. 

The third grid deals with inferential statistics, instances in 
which a generalization about the relationship between the 



9 



o 



numbers generated by observation of some sample are indicative 
of observations that could be made on the entire population. The 
categorizing elements on this grid are the number of dependent 
and independent variables, the level of measurement, and the 
number of groups. Again the determination of the appropriate 
level on each category for a given set of data leads to the 
recommended statistic. 

The use of these grids leads to a specific statistic (in the 
inference grid there is the possibility of alternatives). Through the 
article the analytic procedure actually used can be identified. 
Two quality categories follow from a comparison of the statistic 
used and the statistic appropriate for the data and purpose of the 
study: first, the statistic used is identical with the statistic 
identified and appropriate; second, they are different. In the 
former the research is profiled as appropriately analyzed; in the 
latter, as inappropriately analyzed. 

PROFJLING SUMMARIZED: When a study has been analyzed 
and profiled, it has been described on the following basis: 

A. Is it (I) a test of a hypothesis, or (2) an answer to an 
empirical question? 

At. If it is a test of a hypothesis, is the strength of 
conclusion: I The hypothesis is very little more credible; 
II more credible; or III very much more credible? 



B. What is the quality of the data generation procedure 
(ntjra,)? 



C. Is the data; (a) appropriately analyzed; or (b) inappropri- 
ately analyzed. 

It should be noted that a. single project may consist of several 
sub studies, each of which may be profiled separately. A deci- 
sional flow chart has been developed for arriving at the profile for 
a given study. It is appended. Your reactions regarding its 
adequacy are welcomed. 

It is believed that through profiling completed research their 
adequacies and inadequacies can be made apparent and can more 
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readily be considered as tire conclusions of the research arc 
weighed in decision situations. One further benefit is seen. 
Studies of such profiles should pinpoint problems that could keep 
research nicthodolopisis busy for years to come. 
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STRUCTURE OF A 
LOGICAL ARGUEMENT 




1 2 3 4 5 



PROFILE 



\ A REPORT / 
FROM THE / 
PROFESSIONAL 
LITERATURE / 



/ 



RESEARCH PROFILING FLOW CHART 
LOGIC 



EXAMINE FOR 
PRESENTATION OF 
DATA 




LABEL: Hs 
STOP: ILLOGICAL 
RELATIONSHIPS 
EXIST 



EXAMINE FOR 
HYPOTHESES: 
INCLUDES <1) 2 OR 
MORE VARIABLES (2) 
PREDICTED RELATION- 
SHIP 



STOP: 

REPORT IS 
EITHER 

INCOMPLETE OR 
NOT RESEARCH ~ 




' SERVATION7 




LABEL: H v 


( no i 




TRUTH OF 




HYP 0 THES/S IS 






VERIFIED 'S**' ' 



SS 



© / A LOGICAL ' 

REFERENT observed 
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NOT'- S HYPOTHESES 
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IDENTIFY: 

1. POPULATION OF 
INTEREST 

2. SAMPLE STUDIED 



RESEARCH PROFILING FLOW CHART 
DATA QUALITY - REPRESENTATIVENESS 



✓ DOI^X 



.THE RETORT* 
✓ DELINEATE THE 
V POPULATION TO 
WHICH THE GEN- 
ERALIZATIONS 
APPLY? 



V YES ' 



IDENTIFY: 






THE UNITS 




LABEL: R s 


STUDIED; 


y/ WAS \ /X 


THE ENTIRE 




THE ENTIRE YES ' 


POPULAHON WAS 




X POPULAHON JA 

X. oTUDUiD? / \ y 


STUDIED. 



IDENTIFY: 

HOW THE UNITS 
STUDIED WERE 
SELECTED’ 



X A REX 
THE STUjJ EPn. 

,, UNITS DESCRIBED > 



( YEsV- 



RANDOMIZA* 
✓HON EMPLOYE] 
< TO SELECT A 
SAMPLE FROM T 
POPULAHON OF 
INTEREST? 




LABEL: R 4 
RANDOM SELECTION 
FROM A SPECIFIED 
POPULAHON WAS 
EMPLO' 



S UNI .IS IJttiSCKlRED v / 

N-IN terms OF THEIR 

DtSmtEUTCON ON V J 
RELEVANT 





WERE _ 

.UNKNOWN UNITS ‘ DELIBERATE 
, MEANS SELECTED BY: SELECTION 



LABEL: Rj 
AN UNIDENTIFIED 
GROUP WAS 
STUDIED. 




LABEL: R 3 
PURPOSIVE SAMPLI NG- 
FROM A SPECIFIED 
POPULAHON WAS 
USED. 
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RESEARCH PROFILING FLOW CHART 
DATA QUALITY - TREATMENT 



IDENTIFY: 

1. TREATMENT 

2. DEGREE OF 
SPECIFICITY 
OF PROCEDURE 




HAS 

Each step 

O F TH E TR EATM ENT 
BEEN SPECIFIED 




I LABEL: Tj 
SOMETHING, UNDE- 
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NEITHER TO THE UNITS__ 
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THE MAJOR /— 
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TMFNT SEATED (JR ( RESEARCH 
WAS A STANDARDIZED \rROCEDUIlE 
PROCEDURE USED? ' 



IDENTIFY:. 

1. THEORETICAL BASES 
FOR TREATMENT 

2. CONTROLS FOR 
MEDIATING VARIABLES 

3. CONTROLS FOR EX- 
TRANEOUS VARIANCES 
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MAJOR. 
» STEPS i 



LABEL: T 2 
THE MAIN FEATURES' 
OF THE TREATMENT 
ARE KNOWN 






LABEL: T 6 
THEORY WAS STATED 
MEDIATING VARI- 
ABLES IN IT St 
OTHERS EXTRANEOUS 
TO IT WERE 
CQNTROl 




-G> 



, TECHNIQUES \ LABEL. T 4 

USED TO DtS-\ THEORY WAS STATED - 

IBUTE MEDIATING l YES BUT MEDIATING I 

»/ V J VARIABLES WERE 1 

r NOT CONTROLLED 




LABEL: T S 
THEORY WAS STAT 
and mediating 
VARIABLES WERE 
CONTROLLED 
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RESEARCH PROFILING FLOW CHART 
DATA QUALITY - MEASUREMENT 
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CHART A 

POPULATION DESCRIPTORS 





distribution 


CENTRAL TENDENCY 


ASPERSION - 


NOMINAL 


FREQUENCY IN 
F.AQI category 


mom: 




ORDINAL 


FREQUENCY IN EACH 
SCALOR POSITION 


MEDIAN 


SEMI-INTER QUARTILE 

RAN-G.E 


INTERVAL/ 

RATIO 


FREQUENCY IN 
EACH INTERVAL 


MEAN 


STANDARD DEVIATION 




W.’ 



ri ; • 




CHART R - MEASURES OF ASSOCIATION 

VARIABLE l* 



CONTINUOUS 



FORCED 

DICHOTOMY 



DICHOTOMY 



CONTINUOUS 



PEARSON r 



BISER IAL r 



POINT BISERIAL r 



FORCED 

DICHOTOMY^ 



BISERIAL r 



TETRACHORIC r 



NONE AVAILABLE 



DICHOTOMY 



POINT BISERIAL r 



(NONE AVAILABLE 
USE CHI SQUARE) 



FOUR FOLD r OR 
PHI COEFFICIENT — 



* INTERVAL DATA 



SPEC^C^ 

RANK DATA SPEARMAN’S RHO 



SPECIAL CASE 

MORE THAN 2 VARIABLES. 



INTERVAL DATA 
ORDINAL DATA 
NOMINAL DATA 



MULTIPLE R 
KENDALLS W 
CONTINGENCY COEFF. C 



CHART C 

INDEPENDENT VAR1ABLI 








Most of the mtuurei shown are located with reference to the 
lowest order of data that should be used with them. One should 
always be able to transform the observed data downward l,e. In* 
tezval can be considered ordinal or nominal, ordinal can be 
considered nominal, 
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INDEPENDENT VARIABLE(S) 






|N0MINAL>1 


ORDINAL 1 


ORDINAL> 1 1 


[interval 1 


INTER VAL> 1 


TOE 

JCE OF CHANGES 
RELATED proportions 
NCE 

.HOJTY OF CORRECT CLASSIFICATION 


SIGN TEST 

median test 

MANN-WI ETNEY U TEST 




| ANALYSIS OF VARIANCE 


HOTELLING'S T ■ 

MAHALAN013S 1 D 2 

FISHERS DISCRIMINANT FUNCTION 




frieemans &way anova 




ANALYSIS OF VARIANCE 
J CORRELATION FACTOR ANALYSIS 
MULTIPLE DISCRIMINANT ANALYSIS 


RAOS Vv 

MULTIPLE DISCRIMINANT FUNCTION 


FRIEDMAN'S 2-WAY AN OVA 
: TEST 


INDEX OF ORDER ASSOCIATION 


ANALYSIS OF VARIANCE 
WITH TREND ANALYSIS 












- 




INDEPENDENT t 
ANALYSIS OF VARIANCE 
ANALYSIS OF COVARIANCE 


ANALYSIS OF VARIANCE 




REGRESSION ANALYSIS 


MULTIPLE REGRESSION ANALYSIS 


OS MULTIPLE DISCRIMINANT FUNCTIOI' 

5 






MULTIPLE REGRESSION ANALYSIS 


CANONICAL CORRELATION 


s located with reference to the 
be used with them.. One should 
b served data downward i,e. In- 
ir nomimi, ordinal can be 






This table is an adaptation of Tatcuoka and Tledeman's 
Table 1 in "Statistics As An Aspect of Scientific Method In 
Research on Teaching** pgs. 154-166, in HANDBOOK OF 
RESEARCH ON TEACHING, N. L. Gut {Ed>; Rsnd McNally, 
Chicago, 1963, 1218pp. 
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RESEARCH PROFILE SHEET 

REPORT TITLE: 

AUTHOR— 



SOURCE: 



STOP The report is either not research or it is an incomplete part of 
the research process. 

1 LOGIC 

Q Answer to an Empirical Question 

H s Stop, Illogical relationship in the test of the hypothesis. 

H nc No conclusion can be reached from this test of the 
hypothesis. 

-Hypothesis is questionable. : ___ 

(Rival hypotheses must be considered a cause of the 
consequents) 

H c Hypothesis is credible. 

(Rival hypotheses may be considered a cause of the con- 
sequents) 

H v Hypothesis is verified. 

(Rival hypotheses cannot be considered as a cause of the 
consequents) 

2 DATA QUALITY - REPRESENTATIVENESS 

Rj An unidentified group of subjects was studied. 

R2 Volunteers were studied. 

R3 Purposive sampling from a specified population established 
the group studied. 

R4 Random selection from a specified population established the 
group studied. 

R5 The entire population was studied. 

3 DATA QUALITY - TREATMENT 

Tj No theory; something undefined happened to the units 

studied. 

Tg No theory; treatment description incomplete, or detailed 
elsewhere. 

T3 No theory; treatment described in detail in the report. 

T4 Theory stated but no controls on variables. 

T5 Theory stated and mediating variables controlled. 

Tg Theory stated, mediating variables controlled, and techniques 
used to distribute possible extraneous variances. 
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H V 




T e 


m 6 




H C 


HS 


t 6 


m 5 






R4 




m 4 




h nc 


*3 


T 3 


m 3 


A 


H S 


R2 


T 2 


m 2 


I 


Q 


R 1 


T 1 


M 1 


M 


1 


2 


3 


4 


5 



4 DATA QUALITY - MEASUREMENT 

Available information indicates instrument is invalid for this 
use. , 

M2 Project Developed instrument with low validity (V), reli- 
ability (R), objectivity (O), or other instrument with no info 
about validity or data source. 

M3 Us^d 'Commercially - Produced -or- Other-Project -Dev eloped- 

instrument with low V.R.Q for this application. 

M4 Used Project Developed instrument or Other-Project Deve- 
loped instrument with moderate V. R, O for this application. 

M5 Used instrument which was Project Developed with high V, 
R, O or Other-Project developed with high_ V,R ,0 or 
Commercially Produced with moderate V, R, O for this 
application. 

Mg Used Commercially Produced instrument with high V. R, O 
for this application. 



5 STATISTICAL ANALYSIS 

A Appropriately analyzed 

I Inappropriately analyzed 

M Missing items - incomplete analysis 



